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Abstract 

The goal of this paper is to analyze an intriguing phenomenon recently discovered in deep networks, namely their 
instability to adversarial perturbations (Szegedy et al, 2014). We provide a theoretical framework for analyzing the 
robustness of classifiers to adversarial perturbations, and show fundamental upper bounds on the robustness of classifiers. 
Specifically, we establish a general upper bound on the robustness of classifiers to adversarial perturbations, and then 
illustrate the obtained upper bound on two practical classes of classifiers, namely the linear and quadratic classifiers. 
In both cases, our upper bound depends on a distinguishability measure that captures the notion of difficulty of the 
classification task. Our results for both classes imply that in tasks involving small distinguishability, no classifier in the 
considered set will be robust to adversarial perturbations, even if a good accuracy is achieved. Our theoretical framework 
moreover suggests that the phenomenon of adversarial instability is due to the low flexibility of classifiers, compared to 
the difficulty of the classification task (captured mathematically by the distinguishability measure). Moreover, we further 
show the existence of a clear distinction between the robustness of a classifier to random noise and its robustness to 
adversarial perturbations. Specifically, the former is shown to be larger than the latter by a factor that is proportional to 
s/d (with d being the signal dimension) for linear classifiers. This result gives a theoretical explanation for the discrepancy 
between the two robustness properties in high dimensional problems, which was empirically observed in Szegedy et al 
(2014) in the context of neural networks. To the best of our knowledge, our results provide the first theoretical work that 
addresses the phenomenon of adversarial instability recently observed for deep networks. We finally show experimental 
results on controlled and real-world data that confirm the theoretical analysis and extends its spirit to more complex 
classification schemes. 


1 Introduction 

State-of-the-art deep networks have recently been shown to be surprisingly unstable to adversarial perturbations (Szegedy 
et al, 2014). Unlike random noise, adversarial perturbations are minimal perturbations that are sought to switch the 
estimated label of the classifier. On vision tasks, the results of Szegedy et al (2014) have shown that perturbations that 
are hardly perceptible to the human eye are sufficient to change the decision of a deep network, even if the classifier has 
a performance that is close to the human visual system. This surprising instability raises interesting theoretical questions 
that we initiate in this paper. What causes classifiers to be unstable to adversarial perturbations? Are deep networks the 
only classifiers that have such unstable behaviour? Is it at all possible to design training algorithms to build deep networks 
that are robust or is the instability to adversarial noise an inherent feature of all deep networks? Can we quantify the 
difference between random noise and adversarial noise? Providing theoretical answers to these questions is crucial in 
order to achieve the goal of building classifiers that are robust to adversarial hostile perturbations. 

In this paper, we introduce a framework to formally study the robustness of classifiers to adversarial perturbations 
in the binary setting. We provide a general upper bound on the robustness of classifiers to adversarial perturbations, 
and then illustrate and specialize the obtained upper bound for the families of linear and quadratic classifiers. In both 
cases, our results show the existence of a fundamental limit on the robustness to adversarial perturbations. This limit 
is expressed in terms of a distinguishability measure between the classes, which depends on the considered family of 
classifiers. Specifically, for linear classifiers, the distinguishability is defined as the distance between the means of the 
two classes, while for quadratic classifiers, it is defined as the distance between the matrices of second order moments of 
the two classes. For both classes of functions, our upper bound on the robustness is valid/or all classifiers in the family 
independently of the training procedure, and we see the fact that the bound is independent of the training procedure as 
a strength. This result has the following important implication: in difficult classification tasks involving a small value 
of distinguishability, any classifier in the set with low misclassification rate is vulnerable to adversarial perturbations. 
Importantly, the distinguishability parameter related to quadratic classifiers is much larger than that of linear classifiers for 
many datasets of interest, and suggests that it is harder to find adversarial examples for mom flexible classifiers. We further 
compare the robustness to adversarial perturbations of linear classifiers to the more traditional notion of robustness to 
random uniform noise. The latter robustness is shown to be larger than the former by a factor of s/d (with d the dimension 
of input signals), thereby showing that in high dimensional classification tasks, linear classifiers can be robust to random 
noise even for small values of the distinguishability. We illustrate the newly introduced concepts and our theoretical results 
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on a running example used throughout the paper. We complement our theoretical analysis with experimental results, and 
show that the intuition obtained from the theoretical analysis also holds for more complex classifiers. 

The phenomenon of adversarial instability has recently attracted a lot of attention from the deep network community. 
Following the original paper (Szegedy et al, 2014), several attempts have been made to make deep networks robust to 
adversarial perturbations (Chalupka et al, 2014; Gu and Rigazio, 2014). Moreover, a distinct but related phenomenon has 
been explored in Nguyen et al (2014). Closer to our work, the authors of (Goodfellow et al, 2015) provided an empirical 
explanation of the phenomenon of adversarial instability, and designed an efficient method to find adversarial examples. 
Specifically, contrarily to the original explanation provided in Szegedy et al (2014), the authors argue that it is the “linear” 
nature of deep nets that causes the adversarial instability. Instead, our paper adopts a rigorous mathematical perspective 
to the problem of adversarial instability and shows that adversarial instability is due to the low flexibility of classifiers 
compared to the difficulty of the classification task. 

Our work should not be confused with works on the security of machine learning algorithms under adversarial attacks 
(Biggio et al, 2012; Barreno et al, 2006; Dalvi et al, 2004). These works specifically study attacks that manipulate the 
learning system (e.g., change the decision function by injecting malicious training points), as well as defense strategies 
to counter these attacks. This setting significantly differs from ours, as we examine the robustness of aflxed classifier 
to adversarial perturbations (that is, the classifier cannot be manipulated). The stability of learning algorithms has also 
been defined and extensively studied in (Bousquet and Elisseeff, 2002; Lugosi and Pawlak, 1994). Again, this notion of 
stability differs from the one studied here, as we are interested in the robustness of fixed classifiers, and not of learning 
algorithms. 

The construction of learning algorithms that achieve robustness of classifiers to data corruption has been an active 
area of research in machine learning and robust optimization (see e.g., Caramanis et al (2012) and references therein). 
For a specific disturbance model on the data samples, the robust optimization approach for constructing robust classifiers 
seeks to minimize the worst possible empirical error under such disturbances (Lanckriet et al, 2003; Xu et al, 2009). It is 
shown that, for many disturbance models, the desired objective function can be written as a tractable convex optimization 
problem. Our work studies the robustness of classifiers from a different perspective; we establish upper bounds on the 
robustness of classifiers independently of the learning algorithms. That is, using our bounds, we can certify the instability 
of a class of classifiers to adversarial perturbations, independently of the learning mechanism. In other words, while 
algorithmic and optimization aspects of robust classifiers have been studied in the above works, we focus on fundamental 
limits on the adversarial robustness of classifiers that are independent of the learning scheme. 

The paper is structured as follows: Sec. 2 introduces the problem setting. In Sec. 3, we introduce a running example 
that is used throughout the paper. We introduce in Sec. 4 a theoretical framework for studying the robustness to adversarial 
perturbations. In the following two sections, two case studies are analyzed in detail. The robustness of linear classifiers (to 
adversarial and random noise) is studied in Sec. 5. In Sec. 6, we study the adversarial robustness of quadratic classifiers. 
Experimental results illustrating our theoretical analysis are given in Section 7. Proofs and additional discussion on the 
choice of the norms to measure perturbations are finally deferred to the appendix. 

2 Problem setting 

We first introduce the framework and notations that are used for analyzing the robustness of classifiers to adversarial 
and uniform random noise. We restrict our analysis to the binary classification task, for simplicity. We expect similar 
conclusions for the multi-class case, but we leave that for future work. Let p denote the probability measure on of the 
data points that we wish to classify, and y{x) G {—1,1} be the label of a point x G The distribution p, is assumed to 
be of bounded support. That is, P^(x G B) = I, with = {a; G : ||a :||2 < M} for some M > 0. We further denote 
by Pi and p_i the distributions of class 1 and class —1 in respectively. Let / : M be an arbitrary classification 

function. The classification rule associated to / is simply obtained by taking the sign of f{x). The performance of a 
classifier / is usually measured by its risk, defined as the probability of misclassification according to p: 

R{f) = IP/x(sign(/(x)) 7 ^ y{x)) 

= Pi]Pmi(/W < > 0), 


where p±i = P;i(y(x) = ±1). 

The focus of this paper is to study the robustness of classifiers to adversarial perturbations in the ambient space 
Given a datapoint x G sampled from p, we denote by Aadv(a:; /) the norm of the smallest perturbation that switches 
the sign' of /: 


Aadv(a;; /) = min |lr ||2 subject to /(x)/(x + r) < 0. (1) 

rGR'* 

Here, we use the £2 norm to quantify the perturbation; we refer the reader to Appendix C for a discussion of the norm 
choice. Unlike random noise, the above definition corresponds to a minimal noise, where the perturbation r is sought to 
flip the estimated label of x. This justifies the adversarial nature of the perturbation. It is important to note that, while x 

'We make the assumption that a perturbation r that satisfies the equality f{x + r) = 0 flips the estimated label of x. 
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Figure 1; Illustration of Aadv(a^; /) and Aunif,e(a;; /). The red line represents the classifier boundary. In this case, the 
quantity Aadv(a^; /) is equal to the distance from x to this line. The radius of the sphere drawn around x is Aunif,e(a;; /). 
Assuming f{x) > 0, observe that the spherical cap in the region below the line has measure e, which means that the 
probability that a random point sampled on the sphere has label +1 is 1 — e. 


Quantity 

Definition 

Dependence 

Risk 

R{f) = ]P/..(sign(/(a;)) ^ y{x)) 

p,y,f 

Robustness to adversarial perturbations 

Padv(/) =E^(Aadv(a:;/)) 

pj 

Robustness to random uniform noise 

punif,e(/*) ~ (^unif,e (^i /*)) 

p,f 


Table 1; Quantities of interest in the paper and their dependencies. 


is a datapoint sampled according to p, the perturbed point x + r is not required to belong to the dataset (i.e., x + r can be 
outside the support of p,). The robustness to adversarial perturbation of / is defined as the average of Aadv(a;; /) over all 
x: 


Padv(/) = E^(Aadv(a;; /)). (2) 

In words, p^dvif) is defined as the average norm of the minimal perturbations required to flip the estimated labels of the 
datapoints. Note that Padvif) is a property of both the classifier / and the distribution p, but it is independent of the 
true labels of the datapoints y} Moreover, it should be noted that padv is different from the margin considered by SVMs. 
In fact, SVM margins are traditionally defined as the minimal distance to the (linear) boundary over all training points, 
while padv is defined as the average distance to the boundary over all training points. In addition, distances in our case are 
measured in the input space, while the margin is defined in the feature space for kernel SVMs. 

In this paper, we also study the robustness of classifiers to random uniform noise, that we define as follows. For a 
given e G [0,1], let 


Aunif.£{a:;/) =max77 (3) 

? 7>0 

s.t. f‘n~ 7 ^%{f{x)f{x + n) < 0) < e, 

where pS denotes the uniform measure on the sphere centered at 0 and of radius p in In words, Aunif,e(x; /) denotes 
the maximal radius of the sphere centered at x, such that perturbed points sampled uniformly at random from this sphere 
are classified similarly to x with high probability. An illustration of Aunif,e(a^;/) and A^dv{x; f) is given in Fig. 1. 
Similarly to adversarial perturbations, the point x + n will lie outside the support of p, in general. Note moreover that 
Aunif e(;c; /) provides an upper bound on Aa(jv(a:; /), for all e. The e-robustness of / to random uniform noise is defined 
by: ’ 


PundAf) = /))• (4) 

We summarize the quantities of interest in Table 1. 

3 Running example 

We introduce in this section a running example used throughout the paper to illustrate the notion of adversarial robustness, 
and highlight its difference with the notion of risk. We consider a binary classification task on square images of size 
Vd X Ad. Images of class 1 (resp. class —1) contain exactly one vertical line (resp. horizontal line), and a small constant 
positive number a (resp. negative number —a) is added to all the pixels of the images. That is, for class 1 (resp. —1) 
images, background pixels are set to a (resp. —a), and pixels belonging to the line are equal to 1 -f a (resp. 1 — a). Fig. 2 
illustrates the classification problem for d = 25. The number of datapoints to classify is N = 2\/d. 

^In that aspect, our definition slightly differs from the one proposed in Szegedy et al (2014), which defines the robustness to adversarial perturbations 
as the average of the norms of the minimal perturbations required to misclassify all datapoints. As our notion of robustness is larger, the upper bounds 
derived in our paper also directly apply for the definition of robustness in Szegedy et al (2014). 
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(a) (b) (c) (d) (e) 



(0 (g) (h) (i) (i) 

Figure 2: (a...e); Class 1 images. Class -1 images. 
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(a) Original (b) fu„ 


('-) /quad 


Figure 3: Robustness to adversarial noise of linear and quadratic classifiers, (a): Original image (d = 4, and a = 
(b,c): Minimally perturbed image that switches the estimated label of (b) fii„, (c) /quad- Note that the difference 
between (b) and (a) is hardly perceptible, this demonstrates that fun is not robust to adversarial noise. On the other hand 
images (c) and (a) are clearly different, which indicates that /quad is more robust to adversarial noise 


Clearly, the most relevant concept (in terms of visual appearance) that permits to separate the two classes is the 
orientation of the line (i.e., horizontal vs. vertical). The bias of the image (i.e., the sum of all its pixels) is also a valid 
concept for this task, as it separates the two classes, despite being much more difficult to detect visually. The class of an 
image can therefore be correctly estimated from its orientation or from the bias. Let us first consider the linear classifier 
defined by 


/iin(a:) = - 1, (5) 

where 1 is the vector of size d whose entries are all equal to 1, and x is the vectorized image, exploits the difference of bias 
between the two classes and achieves a perfect classification accuracy for all a > 0. Indeed, a simple computation gives 
= \/da (resp. /iin(a:) = —s/da) for class 1 (resp. class —1) images. Therefore, the risk of /hu is i?(/iin) = 0. It is 
important to note that /un only achieves zero risk because it captures the bias, but fails to distinguish between the images 
based on the orientation of the line. Indeed, when a = 0, the datapoints are not linearly separable. Despite its perfect 
accuracy for any a > 0, fun is not robust to small adversarial perturbations when a is small, as a minor perturbation of 
the bias switches the estimated label. Indeed, a simple computation gives Padv(/iin) = Vda; therefore, the adversarial 
robustness of fun can be made arbitrarily small by choosing a to be small enough. More than that, among all linear 
classifiers that satisfy R{f) — 0, /lin is the one that maximizes Padv(/) (as we show later in Section 5). Therefore, all 
zero-risk linear classifiers are not robust to adversarial perturbations, for this classification task. 

Unlike linear classifiers, a more flexible classifier that correctly captures the orientation of the lines in the images 
will be robust to adversarial perturbation, unless this perturbation significantly alters the image and modifies the direction 
of the line. To illustrate this point, we compare the adversarial robustness of f\i„ to that of a second order polynomial 
classifier /quad that achieves zero risk in Fig. 3, for d = 4? While a hardly perceptible change of the image is sufficient to 
switch the estimated label for the linear classifier, the minimal perturbation for /quad is one that modifies the direction of 
the line, to a great extent. 

The above example highlights several important facts, which are summarized as follows: 

• Risk and adversarial robustness are two distinct properties of a classifier. While R{fiin) = 0, /hu is definitely 
not robust to small adversarial perturbations.^ This is due to the fact that /ii„ only captures the bias in the images 
and ignores the orientation of the line. 

• To capture orientation (i.e., the most visual concept), one has to use a classifier that is fiexible enough for the 
task. Unlike the class of linear classifiers, the class of polynomial classifiers of degree 2 correctly captures the line 
orientation, for d = 4. 

• The robustness to adversarial perturbations provides a quantitative measure of the strength of a concept. 

Since Padv(/im) ^ Padv(/quad), One Can confidently say that the concept captured by /quad is stronger than that of 

^We postpone the detailed analysis of /quad to Section 6. 

''The opposite is also possible, since a constant classifier (e.g., f{x) = 1 for all x) is clearly robust to perturbations, but does not achieve good 
accuracy. 
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/lin, in the sense that the essence of the classification task is captured by /quad, but not by fun (while they are equal 
in terms of misclassification rate). In general classification problems, the quantity Padvif) provides a natural way to 
evaluate and compare the learned concept; larger values of Padv(/) indicate that stronger concepts are learned, for 
comparable values of the risk. 

As illustrated in the above toy example, the robustness to adversarial perturbations is key to assess the strength of a 
concept. In real-world classification tasks, weak concepts correspond to partial information about the classification task 
(which are possibly sufficient to achieve a good accuracy), while strong concepts capture the essence of the classification 
task. 


4 Upper limit on the adversarial robustness 

We now introduce our theoretical framework for analyzing the robustness to adversarial perturbations. We first present a 
key assumption on the classifier / for the analysis of adversarial robustness. 

Assumption (A). There exist r > 0 and 0 < 7 < 1 such that, for all x G B, 

dist{x,S-) < rmax(0,/(x))'^, 
dist(a:, 5'+) < r max(0, —/(a;))'’', 

where dist(a:, 5) = miny{||a; — 2 /II 2 '■ y G S} and S’-!- (resp. S'-) is the set of points x such that/(a;) > 0 (resp. f{x) < 0): 

= {a; : fix) > 0}, 

S_ = {x : fix) < 0}. 

In words, the assumption (A) states that for any datapoint x, the residual max(0, fix)) (resp. max(0, — fix))) can be 
used to bound the distance from x to a datapoint y classified —1 (resp. 1). 

Bounds of the form Eq. (6) have been established for various classes of functions since the early of work of Lo- 
jasiewicz (Lojasiewicz, 1961) in algebraic geometry and have found applications in areas such as mathematical optimiza¬ 
tion (Pang, 1997; Lewis and Pang, 1998). For example, Lojasiewicz (Lojasiewicz, 1961) and later (Luo and Pang, 1994) 
have shown that, quite remarkably, assumption (A) holds for the general class of analytic functions. In (Ng and Zheng, 
2003), (A) is shown to hold with 7 = 1 for piecewise linear functions. In (Luo and Luo, 1994), error bounds on polyno¬ 
mial systems are studied. Proving inequality (6) with explicit constants r and 7 for different classes of functions is still an 
active area of research (Li et al, 2014). In Sections 5 and 6, we provide examples of function classes for which (A) holds, 
and explicit formulas for the parameters t and 7. 

The following result establishes a general upper bound on the robustness to adversarial perturbations; 

Lemma 4.1. Let / be an arbitrary classifier that satisfies (A) with parameters (r, 7). Then, 

Padv(/) < 41 -^T (piE^,(/{x)) -p_iE^_,(/{x)) +2||/|iooi?(/))^ . 

The proof can be found in Appendix A. 1 . The above result provides an upper bound on the adversarial robustness 
that depends on the risk of the classifier, as well as a measure of the separation between the expectations of the classifier 
values computed on distribution pi and p_i. This result is general, as we only assume that / satisfies assumption (A). In 
the next two sections, we apply Lemma 4.1 to two classes of classifiers, and derive interpretable upper bounds in terms of 
a distinguishibality measure (that depends only on the dataset) which quantifies the notion of difficulty of a classification 
task. Studying the general result in Lemma 4.1 through two practical classes of classifiers shows the implications of such 
a fundamental limit on the adversarial robustness, and illustrates the methodology for deriving class-specific and practical 
upper bounds on adversarial robustness from the general upper bound. 


5 Robustness of linear classifiers to adversarial and random perturbations 

The goal of this section is two-fold; first, we specialize Lemma 4.1 to the class of linear functions, and derive interpretable 
upper bounds on the robustness of classifiers to adversarial perturbations (Section 5.1). Then, we derive a formal relation 
between the robustness of linear classifiers to adversarial robustness, and the robustness to random uniform noise (Section 
5.2). 

5.1 Adversarial perturbations 

We define the classification function /(x) = w'^x + b. Note that any linear classifier for which |6| > M||r (;||2 is a trivial 
classifier that assigns the same label to all points, and we therefore assume that |&| < M||r(;|| 2 . 

We first show that the family of linear classifiers satisfies assumption (A), with explicit parameters r and 7. 
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Figure 4; Adversarial robustness versus risk diagram for linear classifiers. Each point in the plane represents a 
linear classifier /. (a): Illustrative diagram, with the non-achievable zone (Theorem 5.2). (b); The exact versus risk 
achievable curve, and our upper bound estimate on the running example. 


Lemma 5.1. Assumption (A) holds for linear classifiers f{x) = w'^x + b with r = l/||t«|j 2 and 7 = 1 . 

Proof. Let x be such that f{x) > 0, and the goal is to prove that dist(a:, S-) < rf^xf^ (the other inequality can be 
handled in a similar way). We have/(a;) = + 6 . dist(a;, S'-) =/(a;)/|ji (;||2 r = l/|ja(;|| 2,7 = 1. ^ 

Using Lemma 4.1, we now derive an interpretable upper bound on the robustness to adversarial perturbations. In 
particular, the following theorem bounds /Oadv{/) from above in terms of the first moments of the distributions pi and 
P-i, and the classifier’s risk: 

Theorem 5.2. Let f{x) = vo^x + h such that | 6 | < M||a(;|| 2 . Then, 

Padv(/) < |biE^i(a;) -p_iE^_ba :)||2 + M{\pi - p-i \ +4i?(/)). (7) 

In the balanced setting where pi = p_i = 1 /2, and if the intercept b = 0 the following inequality holds: 

Padv(/) < ^\\Efj,^{x) -Ef,_^{x )\\2 + 2MR{f). (8) 


Proof. Using Lemma 4.1 with r = 1 /||m ;||2 and 7 = 1, we have 

Padv(/) < (w^ (piE^i (®) - P-lE^-l (*)) + Kpi - p-l) + 2||/||ooi?(/)) (9) 

Observe that 

i. (a:) — (x)) < ||w|| 2 ||piE^j(x)—p_iE^_^(x )||2 using Cauchy-Schwarz inequality. 

ii. h ( p \ — p-i) < M||w||2|pi — P-i| using the assumption |&| < M||w|| 2 , 
iii- ll/lloo = max^,||^|| 2 <M{|w^a: + 6 |} < 2 M||w|| 2 . 

By plugging the three inequalities in Eq. (9), we obtain the desired result in Eq. (7). 

When pi = p_i = 1/2, and the intercept 6 = 0, inequality (iii) can be tightened to |j/||oo < M||m|| 2 , and directly leads to the 
stated result Eq. ( 8 ). □ 

Our upper bound on Padv(/) depends on the difference of means ||E^j (x) — E^ j^ ( 2 ;)|| 2 , which measures the distin- 
guishability between the classes. Note that this term is classifier-independent, and is only a property of the classification 
task. The only dependence on / in the upper bound is through the risk R{f). Thus, in classification tasks where the means 
of the two distributions are close (i.e., |jE^j (x) — E^_j (x )||2 is small), any linear classifier with small risk will necessarily 
have a small robustness to adversarial perturbations. Note that the upper bound logically increases with the risk, as there 
clearly exist robust linear classifiers that achieve high risk (e.g., constant classifier). Lig. 4 (a) pictorially represents the 
Padv vs R diagram as predicted by Theorem 5.2. Each linear classifier is represented by a point on the Pady-R diagram, 
and our result shows the existence of a region that linear classifiers cannot attain. 

Quite importantly, in many interesting classification problems, the quantity ||E^j^ (x) — E^_j (x)|j 2 is small due to large 
intra-class variability (e.g., due to complex intra-class geometric transformations in computer vision applications). There¬ 
fore, even if a linear classifier can achieve a good classification performance on such a task, it will not be robust to small 
adversarial perturbations. In simple tasks involving distributions with significantly different averages, it is likely that there 
exists a linear classifier that can separate correctly the classes, and have a large robustness to adversarial perturbations. 
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Figure 5: Adversarial robustness and robustness to random uniform noise of fun versus the dimension d. We used 
e = 0.01, and a = 0.1/y/d. The lower bound is given in Eq. (10), and the upper bound is the first inequality in Eq. (11). 

5.2 Random uniform noise 

We now examine the robustness of linear classifiers to random uniform noise. The following theorem compares the 
robustness of linear classifiers to random uniform noise with their robustness to adversarial perturbations. 

Theorem 5.3. Let f{x) = vo^x + b. Eor any e G [0,1/12), we have the following bounds on Punif,e(/): 

Punif,e(/) > max (Ci{e)Vd, l) Padv(/), (10) 

Punif,e(/) < C'2(e,d)padv(/) < C 2 (c) V^Padv(/) , (H) 

with C'i(e) = (21n(2/e))-i/2, C^(e, d) = (1 - ( 12 e)^/‘^)-i /2 and C 2 (e) = (1 - 

The proof can be found in appendix A.2. In words, Punif,e(/) behaves as v//padv(/) for linear classifiers (for constant 
e). Linear classifiers are therefore more robust to random noise than adversarial perturbations, by a factor of Vd. In 
typical high dimensional classification problems, this shows that a linear classifier can be robust to random noise even if 
||E^j (x) — E^_^(a ;)||2 is small. Note moreover that our result is tight for e = 0, as we get Punif.o(/) = Padv(/)- 

Our results can be put in perspective with the empirical results of Szegedy et al (2014), that showed a large gap 
between the two notions of robustness on neural networks. Our analysis provides a confirmation of this high dimensional 
phenomenon on linear classifiers. 

5.3 Illustration of the results on the running example 

We now illustrate our theoretical results on the example of Sections. In this case, we have ||E^j^(a:)—E^_j^(a :)||2 = 2y/da. 
By using Theorem 5.2, any zero-risk linear classifier satisfies Padv(/) < Vda. As we choose a IjVd, accurate linear 
classifiers are therefore not robust to adversarial perturbations for this task. We note that /lin (defined in Eq.(5)) achieves 
the upper bound and is therefore the most robust accurate linear classifier one can get, as it can easily be checked that 
Padv(/iin) = Vda. In Eig. 4 (b) the exact p^dv vs R curve is compared to our theoretical upper bound^, for d = 25, iV = 10 
and a bias a = 0.1 /Vd. Besides the zero-risk case where our upper bound is tight, the upper bound is reasonably close to 
the exact curve for other values of the risk (despite not being tight). 

We now focus on the robustness to uniform random noise of fun- Eor various values of d, we compute the upper and 
lower bounds on the robustness to random uniform noise (Theorem 5.3) of /un, where we fix e to 0.01. In addition, we 
compute a simple empirical estimate Punif,e of the robustness to random uniform noise of (see Sec. 7 for details on the 
computation of this estimate). The results are illustrated in Eig. 5. While the adversarial noise robustness is constant with 
the dimension (equal to 0.1, as Padv(/iin) = Vda and a = 0.1/-\/d), the robustness to random uniform noise increases 
with d. Eor example, for d = 2500, the value of punif.e is at least 15 times larger than the adversarial robustness padv- In 
high dimensions, a linear classifier is therefore much more robust to random uniform noise than adversarial noise. 

6 Adversarial robustness of quadratic classifiers 

In this section, we derive specialized upper bounds on the robustness to adversarial perturbations of quadratic classifers 
using Lemma 4.1. 

^The exact curve is computed using a bruteforce approach. 
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6.1 Analysis of adversarial perturbations 

We study the robustness to adversarial perturbations of quadratic classibers of the form f{x) = Ax, where A is a 
symmetric matrix. Besides the practical use of quadratic classifiers in some applications (Goldberg and Elhadad, 2008; 
Chang et al, 2010), they represent a natural extension of linear classifiers. The study of linear vs. quadratic classifiers 
provides insights into how adversarial robustness depends on the family of considered classifiers. Similarly to the linear 
setting, we exclude the case where / is a trivial classiher that assigns a constant label to all datapoints. That is, we assume 
that A satisfies 


'^min {A)< 

0? '^max (A) > 0, 


( 12 ) 


where Amin (A) Amax(A) are the smallest and largest eigenvalues of A. We moreover impose that the eigenvalues of 
A satisfy 


max 


( 

Amin (A) 


'^max(^) 


1 

'^max(^) 


Amin (A) 

) 


<K, 


(13) 


for some constant value K >\. This assumption imposes an approximate symmetry around 0 of the extremal eigenvalues 
of A, thereby disallowing a large bias towards any of the two classes. 

We first show that the assumption (A) is satisfied for quadratic classifiers, and derive explicit formulas for r and 7. 

Lemma 6.1. Assumption (A) holds for the class of quadratic classifiers f(x) = where Amin(^) < 0, Amax(^) > 0 
withr = max(|Amin(A)|"^/^, |Amax(A)|"^/^), and 7 = 1/2, 


Proof. Let x be such that f{x) > 0, and the goal is to prove that dist(a:, S-) < Tf{x)'^ (the other inequality can be 
handled in a similar way). Assume without loss of generality that A is diagonal (this can be done using an appropriate 
change of basis). Let v = — Amin(A). We have f{x) = YltZl ~ 13y setting n = 0 for alH G {1,..., d — 1} 

and Td = sign(a:d) f{x) jv, (where sign(a;) = 1 if a; > 0 and —1 otherwise) we have 


d-\ 

f{x + r) = ^ X.xf - iy{xd + sgn{xd) f{x)/iyf 

= f{x) - 2vxdSg^{xd) \J!(x)lv - f{x) 

= -‘2iy\xd\^/fi,x)/i' < 0 . 


Hence, dist(x, S'-) < ||r||2 = f{x) => t = v ^/^,7=1/2. 


□ 


The following result builds on Lemma 4.1 and bounds the adversarial robustness of quadratic classifiers as a function 
of the second order moments of the distribution and the risk. 

Theorem 6.2. Let f{x) = x"^Ax, where A satisfies Lqs. (12) and (13). Then, 

Padv(/) < 2^K\\p^Ci - + 2MKR(f ), 

where C±i{i,j) = {xiXj))i<i^j<d, and || • ||* denotes the nuclear norm defined as the sum of the singular values of 

the matrix. 

Proof. The class of classifiers under study satisfies assumption (A) with r = max(|Ainin (a)|-vMa max (A)| ^^^), and 7 = 1/2 
(see Lemma 6 . 1 ). By applying Lemma 4 . 1 , we have 

Padv(/) < 2r Ax) - E^_^(x^Aa;) + 2||/||ooi?(/)^ 

Observe that 

i. p_iE^_j(x^Aa;) = (AjPiGi—p_iG_i) < ||A||||piGi—p_iC-i||» usingthegeneralizedCauchy-Schwarz 
inequality, where || ■ || and |j • ||* denote respectively the spectral and nuclear matrix norms. 

ii. |/(®)| = |®^Aa:|<||A||||*||<||A||M, 

iii. |iA||i/V = max(|An,i„(A)|,|A,,ax(A)|)i/2max(|A^i„(A)|-i/MA,nax(A)|-i/2) < 

Applying these three inequalities, we obtain 

padv(/) < 2|!A|//V (llpiCi -p_iG-i||. + 2MR{f)f/^ < 2yA(||piCi -p-iC-ill. + 2MR{f)f/^ . 

□ 


In words, the upper bound on the adversarial robustness depends on a distinguishability measure, defined by ||Ci — 
C_i||*, and the classifier’s risk. In difficult classification tasks, where HCi — C-i ||» is small, any quadratic classifier with 
low risk that satisfies our assumptions in Lq. (12, 13) is not robust to adversarial perturbations. 

It should be noted that, while the distinguishability is measured with the distance between the means of the two distri¬ 
butions in the linear case, it is defined here as the difference between the second order moments matrices IjCi — C-iH*. 
Therefore, in classification tasks involving two distributions with close means, and different second order moments, any 
zero-risk linear classifier will not be robust to adversarial noise, while zero-risk and robust quadratic classifiers are a priori 
possible according to our upper bound in Theorem 6.2. This suggests that robustness to adversarial perturbations can be 
larger for more flexible classifiers, for comparable values of the risk. 
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6.2 Illustration of the results on the running example 

We now illustrate our results on the running example of Section 3, with d = 4. In this case, a simple computation gives 
11 Cl — C_i||* = 2 + 8a > 2. This term is significantly larger than the difference of means (equal to 4a), and there is 
therefore hope to have a quadratic classifier that is accurate and robust to small adversarial perturbations, according to 
Theorem 6.2. In fact, the following quadratic classifier 


/quad(a:) = a;iX2 + Xsx^ - X1X3 - X2Xi, 

outputs 1 for vertical images, and —1 for horizontal images (independently of the bias a). Therefore, /quad achieves 
zero risk on this classification task, similarly to fwn. The two classihers however have different robustness properties to 
adversarial perturbations. Using straightforward calculations, it can be shown that Padv(/quad) = 1 /a/ 2 , for any value 
of a (see Appendix B for more details). For small values of a, we therefore get Padv(/iin) ^ Padv(/quad)- This result is 
intuitive, as /quad differentiates the images from their orientation, unlike /lin that uses the bias to distinguish them. The 
minimal perturbation required to switch the estimated label of /quad is therefore one that modifies the direction of the line, 
while a hardly perceptible perturbation that modifies the bias is enough to flip the label for /quad- This explains the result 
originally illustrated in Fig. 3. 

7 Experimental results 

7.1 Setting 

In this section, we illustrate our results on practical classification examples. Specifically, through experiments on real data, 
we seek to confirm the identified limit on the robustness of classifiers, and we show the large gap between adversarial 
and random robustness on real data. We also study more general classifiers to suggest that the trends obtained with our 
theoretical results are not limited to linear and quadratic classifiers. 

Given a binary classifier /, and a datapoint x, we use an approach close to that of Szegedy et al (2014) to approximate 
^adv(a;; /). Specifically, we perform a line search to find the maximum c > 0 for which the minimizer of the following 
problem satisfies f{x)f{x + r) < 0: 


minc||r||2 + L{f{x + r)sign(/(a;))), 

r 

where we set L{x) = max(0,a;). The above problem (for c fixed) is solved with a subgradient procedure, and we 
denote by A^ 3 y{x] f) the obtained solution.® The empirical robustness to adversarial perturbations is then defined by 
Padv(/) = ^ lyiLi ^adv{a^i; /)> where xi,..., Xm denote the training points. To evaluate the robustness of /, we compare 
Padv(/) to the following quantity: 


K = 


1 

m 


m 

y min 

r-v{xj)^v(xi) 


WXi-Xjh- 


(14) 


It represents the average norm of the minimal perturbation required to “transform” a training point to a training point of the 
opposite class, and can be seen as a distance measure between the two classes. The quantity k therefore provides a baseline 
for comparing the robustness to adversarial perturbations, and we say that / is not robust to adversarial perturbations when 
Padv(/) tt. We also compare the adversarial robustness of the classifiers with their robustness to random uniform noise. 
We estimate Aunif,e(a:; /) using a line search procedure that finds the largest 77 for which the condition 

j#{l < j < J ■ f{x + nj)f{x) < 0} < e, 

is satisfied, where ni,..., nj are iid samples from the sphere 77S. By calling this estimate Aunif,e(a:; /), the robustness of 
/ to uniform random noise is the empirical average over all training points, i.e., Punif,e(/) = — Silli ^unif,e(;Ci; /)■ In 
the experiments, we set J = 500, and e = 0.01.^ 


7.2 Binary classification using SVM 

We perform experiments on several classihers: linear SVM (denoted L-SVM), SVM with polynomial kernels of degree q 
(denoted poly-SVM (q)), and SVM with RBF kernel with a width parameter cr^ (RBF-SVM(a^)). To train the classihers, 

®This procedure is not guaranteed to provide the optimal solution (for arbitrary classifiers /), as the problem is clearly non convex. Strictly speaking, 
the optimization procedure is only guaranteed to provide an upper bound on Aadv(^; /)• 

^We compute the robustness to uniform random noise of all classifiers, except RBF-SVM, as this classifier is often asymmetric, assigning to one of 
the classes “small pockets” in the ambient space, and the rest of the space is assigned to the other class. In these cases, the robustness to uniform random 
noise can be equal to infinity for one of the classes, for a given e. 
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we use the efficient Liblinear (Fan et al, 2008) and LibSVM (Chang and Lin, 2011) implementations, and we fix the 
regularization parameters using a cross-validation procedure. 

We first consider a classification task on the MNIST handwritten digits dataset (LeCun et al, 1998). We consider a 
digit “4” vs. digit “5” binary classification task, with 2, 000 and 1,000 randomly chosen images for training and testing, 
respectively. In addition, a small random translation is applied to all images, and the images are normalized to be of 
unit Euclidean norm. Table 2 reports the accuracy of the different classifiers, and their robustness to adversarial and 
random perturbations. Despite the fact that L-SVM performs fairly well on this classification task (both on training and 
testing), it is highly non robust to small adversarial perturbations. Indeed, padv(/) is one order of magnitude smaller 
than K = 0.72. Visually, this translates to an adversarial perturbation that is hardly perceptible. The instability of the 
linear classifier to adversarial perturbations is not surprising in the light of Theorem 5.2, as the distinguishability term 
^||E^j(a;) — E^_j^(a ;)||2 is small (see Table 4). In addition to improving the accuracy, the more flexible classifiers are 
also more robust to adversarial perturbations, as predicted by our theoretical analysis. That is, the third order classifier is 
slightly more robust than the second order one, and RBF-SVM with small width =0.1 is more robust than with = 1. 
Note that a controls the flexibility of the classifier in a similar way to the degree in the polynomial kernel. Interestingly, 
in this relatively easy classification task, RBF-SVM(O.l) achieves both a good performance, and a high robustness to 
adversarial perturbations. Concerning the robustness to random uniform noise, the results in Table 2 confirm the large gap 
between adversarial and random robustness for the linear classifier, as predicted by Theorem 5.3. Moreover, the results 
suggest that this gap is maintained for polynomial SVM. Fig. 6 illustrates the robustness of the different classifiers on an 
example image. 


Model 

Train error (%) 

Test error (%) 

Padv 

Punif.E 

L-SVM 

4.8 

7.0 

0.08 

0.97 

poly-SVM(2) 

0 

1 

0.19 

2.15 

poly-SVM(3) 

0 

0.6 

0.24 

2.51 

RBF-SVM(l) 

0 

1.1 

0.16 

- 

RBF-SVM(O.l) 

0 

0.5 

0.32 

- 


Table 2: Training and testing accuracy of different models, and robustness to adversarial noise for the MNIST task. Note 
that for this example, we have k = 0.72. 


V y y y y 


(a) 


(b) = 0.08 (c) = 0.19 (d) Aadv = 0.21 (e) A,d, = 0.15 (f) A,dv = 0.41 (g) = 0.8 


Figure 6: Original image (a) and minimally perturbed images (b-f) that switch the estimated label of linear (b), quadratic 
(c), cubic (d), RBF(l) (e), RBF(O.l) (f) classifiers. The image in (g) corresponds to the original image perturbed with a 
random uniform noise of norm Aunif,e(a;; /), where / is the learned linear classifier. That is, the linear classifier gives the 
same label to (a) and (g), with high probability. The norms of the perturbations are reported in each case. 

We now turn to a natural image classification task, with images taken from the CIFAR-10 database (Krizhevsky and 
Hinton, 2009). The database contains 10 classes of 32 x 32 RGB images. We restrict the dataset to the first two classes 
(“airplane” and “automobile”), and consider a subset of the original data, with 1, 000 images for training, and 1,000 for 
testing. Moreover, all images are normalized to be of unit Euclidean norm. Compared to the first dataset, this task is more 
difficult, as the variability of the images is much larger than for digits. We report the results in Table 3. It can be seen 
that all classifiers are not robust to adversarial perturbations for this experiment, as Padv(/) ^ k = 0.39. Despite that, 
all classifiers (except L-SVM) achieve an accuracy around 85%, and a training accuracy above 92%, and are robust to 
uniform random noise. Fig. 7 illustrates the robustness to adversarial and random noise of the learned classifiers, on an 
example image of the dataset. Compared to the digits dataset, the distinguishability measures for this task are smaller (see 
Table 4). Our theoretical analysis therefore predicts a lower limit on the adversarial robustness of linear and quadratic 
classifiers for this task (even though the bound for quadratic classifiers is far from the achieved robustness of poly-S VM(2) 
in this example). 

The instability of all classifiers to adversarial perturbations on this task suggests that the essence of the classification 
task was not correctly captured by these classifiers, even if a fairly good test accuracy is reached. To reach better robust¬ 
ness, two possibilities exist: use a more flexible family of classifiers (as our theoretical results suggest that more flexible 
families of classifiers achieve better robustness), or use a better training algorithm for the tested nonlinear classifiers. The 
latter solution seems possible, as the theoretical limit for quadratic classifiers suggests that there is still room to improve 
the robustness of these classifiers. 
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Model 

Train error (%) 

Test error (%) 

Padv 

Punif,e 

L-SVM 

14.5 

21.3 

0.04 

0.94 

poly-SVM(2) 

4.2 

15.3 

0.03 

0.73 

poly-SVM(3) 

4 

15 

0.04 

0.89 

RBF-SVM(l) 

7.6 

16 

0.04 

- 

RBF-SVM(O.l) 

0 

13.1 

0.06 

- 


Table 3: Training and testing accuracy of different models, and robustness to adversarial noise for the CIFAR task. Note 
that for this example, we have k = 0.39. 



(a) (b) Aadv = 0.04 (c) = 0.02 (d) A^dv = 0.03 (e) A^dv = 0.03 (f) A^dv = 0.05 (g) = 0.8 

Figure 7: Same as Fig. 6, but for the “airplane” vs. “automobile” classification task. 


7.3 Multiclass classification using CNN 

Since our theoretical results suggest that more flexible classifiers achieve better robustness to adversarial perturbations in 
the binary case, we now explore empirically whether the same intuitions hold in scenarios that depart from the theory in 
two different ways: (i) we consider multiclass classification problems, and (ii) we consider convolutional neural network 
architectures. While classifiers’ flexibility is relatively well quantified for polynomial classifiers by the degree of the 
polynomials, this is not straightforward to do for neural network architectures. In this section, we examine the effect of 
breadth and depth on the robustness to adversarial perturbations of classifiers. 

We perform experiments on the multiclass ClFAR-10 classification task, and use the recent method in (Moosavi- 
Dezfooli et al, 2016) to compute adversarial examples in the multiclass case. We focus on baseline CNN classifiers, 
and learn architectures with 1, 2 and 3 hidden layers. Specifically, each layer consists of a successive combination 
of convolutional, rectified linear units and pooling operations. The convolutional layers consist of 5 x 5 Alters with 
50 feature maps for each layer, and the pooling operations are done on a window of size 3x3 with a stride parameter 
of 2. We build the three architectures gradually, by successively stacking a new hidden layer on top of the previous 
architecture (kept fixed). The last hidden layer is then connected to a fully connected layer, and the softmax loss is used. 
All architectures are trained with stochastic gradient descent. To provide a fair comparison of the different classifiers, all 
three classifiers have approximately similar classification error (35%). To ensure similar accuracies, we perform an early 
stop of the training procedure when necessary. The empirical normalized robustness to adversarial perturbations of the 
three networks are compared in Figure 8 (a).^ 

We observe first that increasing the depth of the network leads to a significant increase in the robustness to adversarial 
perturbations, especially from 1 to 2 layers. The depth of a neural network has an important impact on the robustness of the 
classifier, just like the degree of a polynomial classifier is an important factor for the robustness. Going from 2 to 3 layers 
however seems to have a marginal effect on the robustness. It should be noted that, despite the increase of the robustness 
with the depth, the normalized robustness computed for all classifiers is relatively small, which suggests that none of these 
classifiers is really robust to adversarial perturbations. Note also that the results in Figure 8 (a) showing an increase of the 
robustness with the depth are inline with recent results showing that depth provides robustness to adversarial geometric 
transformations (Fawzi and Frossard, 2015). In Fig. 8 (b), we show the effect of the number of feature maps in the 
CNN (for a one layer CNN) on the estimated normalized robustness to adversarial perturbations. Unlike the effect of 
depth, we observe that the number of feature maps has barely any effect on the robustness to adversarial perturbations. 
Finally, a comparison of the normalized robustness measures of very deep networks VGG-16 and VGG-19 (Simonyan and 
Zisserman, 2014) on ImageNet shows that these two networks behave very similarly in terms of robustness (both achieve 
a normalized robustness of 3 • 10“^). This experiment, along with the experiment in Figure 8 (a), empirically suggest that 
adding layers on top of shallow network helps in terms of adversarial robustness, but if the depth of the network is already 
sufficiently large, then adding layers only moderately changes that robustness. 

8 Discussion and perspectives 

The existence of a general limit on the adversarial robustness of classifiers (established in Lemma 4.1) is an important 
phenomenon with many practical implications. To better understand the implications of this limit, we derived specialized 
upper bounds for two families of classifiers. For the family of linear classifiers, the established limit is very small for 

*More precisely, we report ^ 
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Quantity 

Definition 

Digits 

Natural images 

Distance between classes 

K (see Eq. (14)) 

0.72 

0.39 

Distinguishability (linear class.) 

||piE^i(a;) -p iE^i(a ;)||2 

0.14 

0.06 

Distinguishability (quadratic class.) 

2 VA'lbiCi -p-iC-ill* 

1.4 

0.87 


Table 4: The parameter k, and distinguishability measures for the two classification tasks. For the numerical computation, 
we used K = 1. 




Number of feature maps 


(a) Evolution with depth 


(b) Evolution with breadth 


Figure 8: Evolution of the normalized robustness of classifiers with respect to (a) the depth of a CNN for CIFAR-10 task, 
and (b) the number of feature maps. 


most problems of interest. Hence, linear classifiers are usually not robust to adversarial noise (even though robustness to 
random noise might be achieved). This is however different for nonlinear classifiers; for the family of quadratic classifiers, 
the limit on adversarial robustness is usually larger than for linear classifiers, which gives hope to have classifiers that 
are robust to adversarial perturbations. In fact, by using an appropriate training procedure, it might be possible to get 
closer to the theoretical bound. For general nonlinear classifiers (e.g., neural networks), designing training procedures 
that specifically take into account the robustness in the learning is an important future work. We also believe that the 
application of our general upper bound in Lemma 4.1 to derive explicit upper bounds that are specific to e.g., deep neural 
networks is an important future work. To do that, we believe that it is important to derive explicitly the parameters (r, 7) 
of assumption (A) for the class of functions under consideration. Results from algebraic geometry seem to suggest that 
establishing such results might be possible for general classes of functions (e.g., piecewise linear functions). In addition, 
experimental results suggest that, unlike the breadth of the neural network, the depth plays a crucial role in the adversarial 
robustness. Identifying an upper bound on the adversarial robustness of deep neural networks in terms of the depth of the 
network would be a great step towards having a better understanding of such systems. 
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A Proofs 


A.l Proof of Lemma 4.1 

We begin by proving the following inequality: 

Lemma A.l. Let zi,... ,Znbe non-negative real numbers, and let 0 < 7 < 1. Then, 

n / ^ \ ^ 

2 = 1 \2=1 / 


Proof. We prove that the quantity 
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is bounded from above by ^. To do so, let Ui = —, and let us determine the maximum of the concave function 

g{ui ,..., Un-i) = uj + ■ ■ ■ + {1 — ui — ■ ■ ■ — Un-i)~^■ Setting the derivative of g with respect to Ui to zero, we get 

uj~^ - (1 - Ml-M„_i)'>'“^ = 0, 


hence Ui = 1 —Mi -Wn-i- We therefore get ui = • • • = m„_i, and conclude that the maximum of z j 

is reached when zi = • • • = and the value of the maximum is □ 

We now prove Lemma 4.1 . 

Proof. The goal is to hnd an upper bound on = E^(Aa(iv(2:; /))• 

/^adv(/) — (Aadv(:t:; /)) T J*—(Aadv(d:; /)) 

= Pi(E^i(Aadv(a;;/)|/(a:) > 0)P^7/(a:) > 0) + E^7Aadv(a:;/)|/(a;) < 0)P^7/(x) < 0)) 

+ p_i(E^_7Aadv(a:;/)|/(a:) < 0)P^_7/(x) < 0) + E^_7Aadv(a;;/)|/(x) > 0)P^_7/{x) > 0)^ 

Using assumption (A), the following upper bounds hold: 


Ei,±i(Aadv(a:;/)|/(a;) > 0) < TEf,^,{f{xy\f{x) > 0) 

Ei,±i(Aadv(a:;/)|/(a;) < 0) < TE^j^^{{-f{x))^\f{x) < 0) 

Hence, we obtain the following inequality on Padv(/): 

Padv(/) < Tpi(E^7/(x)7/(a:) > 0)P^7/(x) > 0)+E^,{{-f{x)r\f{x) < 0)P^7/(x) < 0)) 

+ Tp_i(E^_^{{-f{x)y\f{x) < 0)Pf,_,if{x) < 0) +E^_^{f{xy\f{x) > 0)P^_7/(a;) > 0)). 

Using Jensen’s inequality, we have E(A^) < E(A)''', for any random variable X, and 7 < 1. Using this inequality 
together with P(A) < P(A)''', we obtain 


Padv(/) < T(^{piE^^{f{x)\fix) > 0)F^^{f{x) > 0))'*' + (piE^7-/(a;)|/(a:) < 0)P^7/(a:) < 0))'^ 

+ {p_,E^_,{-fix)\f{x) < 0)P^_7/(a:) < 0))^ + (p_iE^_7/(x)|/(a:) > 0)P^_7/(a:) > 0))^). 

We use the result in Lemma A. 1 with n = 4, and obtain 

Padv(/) < T4^-'^(piE^7/(a:)|/(x) > 0)F^,{f{x) > 0) + piEf,^{-f{x)\f{x) < 0)F^^{f{x) < 0) 

+ P-iE^_,{-f{x)\f{x) < 0)F^_,{f{x) < 0)+p-iE^_,{f{x)\f{x) > 0)P^_7/(a;) > 0))^. 

Note moreover that the following equality holds 

-PilP/xi(/(a:) < 0)E^7/(a;)|/(a:) < 0) 

= 2piP^i(/(x) < 0)|E^7/(a;)|/(a;) < 0)| +piP^7/(x) < 0)E^^{f{x)\f{x) < 0), 
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Using the above equality along with a similar one for (/(a;) > 0)Efj__^{f{x)\f{x) > 0), the following upper 

bound is obtained 

Padv(/) < (piEf,,{f{x)\f{x) > 0)F^,{f{x) > 0) + piEi,,{f{x)\f{x) < 0)P^i(/(x) < 0) 

-p-iE^_Af(.x)\f{x) < 0)P^_,(/(a;) < 0) - (/(a:)|/(x) > 0)P^_,(/(a:) > 0) 

+ 2piP^i(/(x) < 0)|E^,(/(a;)|/(a:) < 0| + 2p_iP^_,(/(x) > 0)|E^_,(/(a;)|/(a;) > 0|))^, 

which simplifies to 

Padv(/) < T4i"'*'(piE^^(/(a;)) -p_iE^_j(/(a;)) + 2piP^j(/(a;) < 0)|E^^(/(a;)|/(a;) < 0)| 

+ 2p_iP^_,(/(x) > 0)|E^_,(/(x)|/(x) > 0)1)^ 

Observe moreover that i?(/) =piP^^(/(x) < 0) + p_iP^_j (/(x) > 0), and that |E^_j (/(x)|/(x) > 0)| is bounded 
from above by ||/||oo- We therefore conclude that 

Padv(/)<r4i-'^(piE^,(/(x))-p_iE^_,(/(x)) + 2i?(/)||/||^)^ 

□ 


A.2 Proof of Theorem 5.3 

The proof of this theorem relies on the concentration of measure on the sphere. The following result from (Matousek, 
2002 ) precisely bounds the measure of a spherical cap. 

Theorem A.2. Let '^(r) = {x S ■ Xi > t} denote the spherical cap of height 1 — r. Then for 0 < r < y^2jd, we 
have ^ < P(‘^(t)) < i, and for ^/2/d < t < 1, we have; 


Based on Theorem A.2, we show the following result: 

Lemma A.3. Let w be a vector of unit £2 norm in M'^. Let r G [0,1), and x be a vector sampled uniformly at random 
from the unit sphere in Then, 


^(1 - tY < > r}) < 2 exp . 

Proof. Using an appropriate change of basis, we can assume that rc = (1,0,..., 0)^. For r G [•\/2/d, 1), we have 

(a) 1 (b) 

P({a^i>T}) < - 2 exp(-r"d/ 2 ), 

where (a) uses the upper bound of Theorem A.2, and (b) uses the inequality (1 — r^) < exp(—r^). Note moreover that 
for r G [0, y/2jd), the inequality 2 exp(—T^d/2) > 2 exp(—1) >1/2 holds, which proves the upper bound. 

We now prove the lower bound. Observe that the following lower bound holds for (r-\/d)“^, for any r G 

-^j= > exp(-T2d/2). 

To see this, note that the maximum of the function a 1 — ln(a) /is equal to 1/(2e) < 1 /2. Therefore, ln{T'/d )/ (r^d) < 
1/2, or equivalently, {T'/d)~^ > exp(—T^(i/2). Therefore, we get > (1 — and using Theorem A.2, we 

obtain for any r G [^/2ld, 1): 


P({xi > r}) > ^(1 - > ^(1 - r^. 

Note also that this inequality holds for t G [0, ^y2fd], as ^(1 — t^)'^ < 


□ 
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Armed with the concentration of measure result on the sphere, we now focus on the proof of Theorem 5.3. Let f{x) = 
iD^x + h. Let X be fixed such that f{x) > 0, and let p > 0 and e € (0,1/12). Then, 

Pn^TjS (/(a: + n) < 0 ) = Pn^r,s (w^n < -nFx - b) 

= Pn^riS {w'^n/\\w\\2 < /)) 

= P„,..s {w'^n/\\w\\2 < -Aadv(a:; f)/v) 

Using the upper bound in Lemma A.3, we obtain: 

Pn-rjS {f{x + n) <0) <2 exp • 

Therefore, for p = (21n(2/e))“^/^v^Aadv(a:;/) = C'i(e)v^Aadv{a;; /), we obtain Vn~r]s{fix + n) < 0) < e, and we 
deduce that 


Aunif.e(a:,/*) ^ (e)v^Aadv(a:,/*). 

Using the lower bound result of Lemma A.3, we have: 

This implies that for any rj > = C 2 {e, d)Aadv(a;; /), we have if{x + ri) < 0) > e. Hence, we obtain 

y 1 —(12e)i/a 

the following upper bound on Aunif,e(a:; /): 

Aunif.e (a:,/) ^ tl) Aadv(a:,/). 


We also derive a lower bound on Aunif,e(a;; /) of the form C 2 (e)-\/dAadv{a:; /) by noting that 

1 1 


C 2 (e, = 

where we have used the fact that 


< 


= C2ie), 


^d{l-{12ey/d) “ Vl - 12 e 
is a decreasing function of d. To see that this function is indeed decreas¬ 


ing, note that its derivative (with respect to d) can be written as P{d) ~ 1) ~ lii(e)), with P{d) non-negative. 


Vd(l-( 12 e)i/a) 

gi/d _ 1 ) _ gi/din(e 

and e = 12e. Then, by using the inequality < {1/e)^^'^ — 1, the negativity of the derivative follows. 

By combining the lower and upper bounds, and taking the expectations on both sides of the inequality, we obtain: 

Ci(e)v/®;i ^Aadv(a:;/)lj(a;)>o) A (Aunif,£(a:;/)lj^(x)>o) — C'2(£;(^)P/i (Aadv(a:;/)lj(a;)>o) 

< C2(e)VdE^ (A 

adv (x', f)lf(x)>o) ■ 

A similar result can be proven for x such that f(x) < 0. We therefore conclude that 

max(C'i(e)v^, l)padv(/) < Punif.cC/) < C 2 (e,d)p^dvif) < C' 2 (e)v^Padv(/), 
where we have used the inequality Punif,e(/) > Padv(/)- 


B Vertical-horizontal example: quadratic classifier 

We consider the quadratic classifier /quad (a:) = x'^ Ax, with 

0 1 -1 0 ■ 

10 0-1 

-10 0 1 ■ 

0-110 

We perform a change of basis, and work in the diagonalizing basis of A, denoted by P. We have 




1 

1 


1 -L 

1 


0 


-72 0 

2 


0 

0 

72 



1 

-1 

1 

- 1 _ 



'1 

0 0 

0 ■ 




0 

0 0 

0 

p. 



0 

0 0 

0 




0 

0 0 

-1 
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By letting x = Px, we have: 


/quad(:^) — :^4- 


Given a point x and label y, the following problem is solved to find the minimal perturbation that switches the estimated 
label: 


minf^ + s.t. y((xi + fi)^ — (a ;4 + f 4 )^) < 0 . 

r 

Let us consider the first datapoint x = [1 + a, 1 + a, a, a]^ (the other points can be handled in an exactly similar fashion). 
Then, it is easy to see that ii = 1 and 2:4 = 0, and the optimal point is achieved for fi = —1/2 and f 4 = 1/2. In 
the original space, this point corresponds to r = = [0, —1/2,1/2, 0]^. Therefore, ||r ||2 = l/-\/2, and we obtain 

Padv(/quad) — l/v/^. 


C Discussion on the norms used to measure the magnitude of adversarial per¬ 
turbations 

The goal of this section is to discuss different ways of measuring the robustness to adversarial perturbations. 

Given a datapoint x, let p > 0 be such that we know a priori that all points in the region 

TZ{x) = {z : N{z — x) < rj}, 

have the same true class as x (i.e., a human observer would classify all images in this region similarly). Here iV : 

M+ defines a norm in the image space. Note that TZ{x) only depends on the dataset, but does not depend on any classifier 
/. We defined the robustness of / to adversarial perturbations, at x, to be 

^adv{a:; /) = min A^(r) subject to /(x + r)f{x) < 0 . 

r 

The classifier / is said to be not robust at x if 


Aadv(x;/)<p. (15) 

In words, this means that there exists a point z in the region TZ{x) (i.e., z and x are classified in the same way by a human 
observer), but z is classified differently than x by /. Our main theoretical result provides upper bounds to Padv(/) (the 
expectation of Aadv(x; /)) in terms of interpretable quantities (i.e., distinguishability and risk): Padv(/) A U{fi, R{f)). 
Using this upper bound and Eq. (15), we certify that / is not robust to adversarial perturbations when the following 
sufficient condition holds: 


U{y,R{f))<y. (16) 

The main difficulty in the above definitions lies in the choice of N and ry: how can {N, rj) be chosen to guarantee 
that TZ{x) contains all images of the same underlying class as x? In the original paper (Szegedy et al, 2014), N is set 
to be the £2 norm, but no p is formally derived; classifiers are said to be not robust to adversarial perturbations when 
PadviD/Vd is judged to be “sufficiently small”. For example, it appears from Table 1 in (Szegedy et al, 2014) that if 
PadviD/Vd < 0 . 1 , the minimum required perturbation is thought to be small enough to guarantee that the images do not 
change their true underlying label. Motivated by the fact that pixels (or features) have limited precision, (Goodfellow et al, 
2015) consider instead the norm, and ideally assume that a perturbation that have iao norm smaller than the precision 
of the pixels (e.g., 1/255 of the dynamic range for 8 -bit images) is guaranteed to conserve the true underlying class. While 
this corresponds to setting rj to be the precision of the pixels, in practice it is set to be much larger for the MNIST case, as 
the images are essentially binary. In our case, the £2 norm is considered, and we define the quantity k to be the average 
norm of the minimal perturbation required to transform a training point to a training point of the opposite class: 


K = 


1 

m 


m 


E 


min 


\\Xi-Xj\\2. 


We assume that the image x -f r is of the same underlying label as x if ||r ||2 is one order of magnitude smaller than k. 
This corresponds to setting rj = k/ 10. A summary of the different choices is shown in Table 5. 

All the above choices represent proxies of what we really would like to capture (i.e., the notion of perceptibility and 
class change). They all have some benefits and drawbacks, which we mention briefly now. We first acknowledge that 
the £00 norm with ry « 0.1 blocks class changes (and therefore provides a sufficient condition for certifying the non¬ 
robustness of classifiers) for images that are essentially binary (e.g, MNIST digit images). In those cases, the £^0 norm 
seems more appropriate to use than the £2 norm. However, in order to compare both norms, we need to carefully (and 
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N 


(Szegedy et al, 2014) 

Mb 

- 

(Goodfellow et al, 2015) 

Mloo 

Determined by the image preci.sion (in theory). Larger in practice. 

Ours 

Mb 

k/10 


Table 5; Different choices of N and rj in different papers. 


r ^ 

L JL JL J 

(a) (b) (c) 

Figure 9: Example images in a toy classification problem where the goal is to distinguish the different balls (a: basketball, 
b; soccer), (c) represents an umbrella that does not belong to any class. Black pixels are equal to 0, white pixels are equal 
to 1, grey pixels are set around 0.9. 


fairly) choose the p parameter for both norms. In fact, if it is acknowledged that N = || • ||oo and 77 = 0.1 provides 
a valid region TZ where underlying image classes do not change, then IV = || • II 2 and 77 = 0.1 also provides a valid 
region, as ||r||oo < ||f ||2 for any vector r. It is therefore all a matter of choosing a right threshold 77 that is fair for all 
norms, if we wish to compare the norms for the task that we have at hand. A comparison between the £00 and £2 norm 
is provided in (Goodfellow, 2015), and it is concluded that, while the £2 norm allows for class changes within its region, 
the £00 essentially blocks the class changes and therefore constitutes a better choice. In more details, the comparison goes 
as follows: it is first argued that by choosing iV = || • II 2 and 77 = 3.96, the region TZ contains both a “natural” 3 and 7, 
and therefore does not provide a valid region. To show the benefits of the £^0 norm, the author proceeds by considering 
^ = II • II 00 and 77 = 3.96/v/d « 0.1414. It is then argued that this region blocks previous attempts for class changes, and 
therefore the £00 norm provides a better choice for the task at hand. While this type of comparison is important in order 
to reach a better understanding of the norms used to measure the adversarial examples, it is not conclusive as it is unfair 
to the £2 norm. Let us recall the following inequalities 

Vr e K"*, ||r||oo < \\r \\2 < VdllT-Hoo- (17) 

For a fixed 770 > 0, define the regions: 

'^00 — • ||-2^ ^||oo ^ ^0}; 

TZ 2 = {z : ||z - a ;||2 < ?7oV^}- 

It should be noted that for any 770 , we have TZao C TZ 2 using Eq. (17). Not only that, but TZcx> constitutes a tiny portion 
of TZ 2 in high dimensional spaces (i.e., the volume of TZao over that of TZ 2 decays exponentially with the dimension). 
Therefore, a comparison of 72-2 to TZoo will typically lead to problematic images in TZ 2 but not in TZoo, as 72-2 is much 
bigger than TZoo- Therefore, the fact that TZoo is a much smaller set than TZ 2 (i.e., it contains much less images) is already 
known from Eq. (17) and is not conclusive in terms of the comparison of the two norms for measuring the robustness 
to adversarial perturbations. Just like the comparison of TZ 2 to TZoo is unfair to the £2 norm, saying that the £2 norm is 
better than the ioo norm because TZoo contains much more images (potentially problematic ones with class changes, for 
sufficiently large 770 ) that are not in TZ 2 = {z : \\z — x \\2 < po} ts unfair to the £00 norm. 

One possible way for providing a fair comparison between both norms is to find the coefficient c such that TZoo has 
the same volume as the following £2 ball 

7^2 = ■ 11 ^ - a ;||2 < with 

Using mathematical derivations that we omit for the flow of this short discussion, we obtain c = ~ 0.48 asymptoti¬ 

cally as d — 00 . We argue that the comparison of TZoo to TZ'-^ provides a more conclusive experiment than comparing TZoo 
to TZ 2 , as it highlights the advantage of one norm with respect to the other without biases on the volume of the region. In 
practice, this new comparison implies the following change for the “3” vs. “7” example in Goodfellow (2015): instead 
of allowing perturbations of max-norm 0.1414, perturbations with £00 norm up to « 0.3 are allowed. This will result 
in images that are roughly twice as much perturbed, for the £00 case. Even with this comparison, it is possible that the 
max-norm in this case will also block attempts to change the class, as the images are essentially binary. We believe that 
the £00 is probably a better choice in this case. 

However, this is not a general statement, as in some cases of non-binary images, the £2 norm might be a better choice. 
We illustrate the above statement on a toy example where the goal is to classify sport balls. Some example images are 
shown in Eig. 9. In this example, the £00 norm between any two images is less than 0.11. Setting 770 = 0.11 with 
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^ = II • Iloo does not define a valid region (i.e., it does not guarantee that no class changes will occur within the region). 
On the other hand, the region 7^2 computed with po = 0.11 (i.e., pg = 0.0532) rightfully excludes the images b) and c) 
from the space of valid perturbations of a). This toy example provides a proof of concept that shows that, in some cases, 
the £2 norm might actually be a better choice than the norm. 

In conclusion, we stress that this example has no intention of proving that the £2 norm is universally better than the 
£ao norm to measure the norm of adversarial perturbations. Through this discussion and example, we show that there is 
no universal answer to which norm one has to use to measure the robustness to adversarial perturbations, as it is strongly 
application-dependent. We believe more theoretical research in that area is needed in order to fully grasp the advantage 
of each norm, and probably design new norms that are suitable for measuring adversarial perturbations. 
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